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Abstract 


More  so  than  engineered  systems,  human  factors,  and  specifically  having  humans-in-the-loop, 
can  lead  to  unforeseen  behaviours  resulting  in  unexpected  organizational  failures.  In  the  world  of 
emergency  response,  these  failures  may  be  related  not  only  to  response  activities,  but  also  to 
information  processing  and  sharing  that  consequently  undermine  the  organizational  ecosystems’ 
situational  awareness  of  unfolding  events.  The  TIF  project,  Modelling  Public  Security 
Operations,  has  the  goal  of  accounting  for  the  human  factor  by  more  fully  exploring  its  inherent 
complexity  through  experiments  and  simulations.  This  work  reviews  the  development  of  a  new 
capability  in  this  area  using  computer  simulation,  as  well  as  the  lessons  learned  from  this  effort.  It 
represents  a  detailed  proof-of-concept  into  how  human  factors  can  be  explored  through  multi¬ 
agent  systems.  The  modeling  and  simulation  challenge  remains  that  more  research  on  human 
factors  must  be  conducted,  including  work  to  support  improved  validation  methods  for  computer 
human-factor  models 


Resume 


Plus  que  les  systemes  sophistiques,  les  facteurs  humains,  et  plus  particulierement  lorsque  des 
humains  interviennent,  peuvent  mener  a  des  comportements  imprevus  entrainant  des  echecs 
organisationnels  imprevus.  Dans  le  monde  des  interventions  d’urgence,  ces  echecs  peuvent  etre 
relies  non  seulement  aux  activites  d’ intervention,  mais  aussi  au  traitement  et  a  Pechange  de 
Tinformation  qui  minent  la  capacite  de  Torganisation  a  jauger  la  situation  a  mesure  qu’elle 
evolue.  Le  projet  FIT,  la  modelisation  des  operations  de  securite  publique,  vise  a  tenir  compte  du 
facteur  humain  en  examinant  plus  amplement  sa  complexity  inherente  au  moyen  d’ experiences  et 
de  simulations.  Le  present  travail  consiste  a  examiner  le  developpement  d’une  nouvelle  capacite 
dans  ce  domaine  au  moyen  d’une  simulation  informatisee  ainsi  que  des  logons  retenues  de  cet 
exercice.  II  valide  le  principe  selon  lequel  les  facteurs  humains  peuvent  etre  etudies  au  moyen  de 
systemes  multi-agents.  Modelisation  et  simulation  ou  pas,  plus  de  recherches  sur  les  facteurs 
humains  doivent  etre  faites,  y  compris  des  travaux  pour  etayer  de  meilleures  methodes  de 
validation  pour  les  modeles  informatisees  des  facteurs  humains. 
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Executive  summary 


Modelling  Public  Security  Operations:  Evaluation  of  the  Holistic 
Security  Ecosystem  (HSE)  Proof-of-Concept 

Alexis  Morris;  William  Ross;  Mihaela  Ulieru;  DRDC  CSS  CR  2012-026;  DRDC 
CSS  December  2012 

Introduction:  This  report  summarizes  and  evaluates  research  performed  by  the  University  of 
New  Brunswick  (UNB)  in  the  context  of  the  DRDC  project,  "Modelling  Public  Security 
Operations ,”  which  aims  to  investigate  decision-making  in  complex  meta-organizations.  The 
UNB  component  of  this  initiative  has  involved  the  design  and  implementation  of  a  simulation 
approach,  the  Holistic  Security  Ecosystem  (HSE)  simulation,  for  representing  organizations  and 
the  decision-making  process  and  for  simulating  the  impact  of  key  social,  cognitive,  and 
informational  factors  on  the  joint  effectiveness  of  a  security  ecosystem. 

Results:  The  Holistic  Security  Ecosystem  (HSE)  portion  of  the  DRDC  project  was  tasked  with 
the  goal  of  examining  joint-organizational  systems  from  both  a  top-down  (i.e.,  individual)  and 
bottom-up  (i.e.,  system)  perspective.  It  focused  on  the  development  of  a  computer-simulation 
approach  to  explore  the  effect  of  key  social,  cognitive,  and  informational  factors  present  in 
individual  agents  (i.e.,  the  individual-level)  on  inter-organizational  efforts  (i.e.,  the  system-level), 
in  particular  consensus  achievement  through  information  sharing.  The  focus  of  the  simulation  was 
on  whether  or  not  agents  with  bounded  rationality,  only  witnessing  portions  of  the  unfolding 
situation,  could,  in  conjunction  with  other  agents  and  based  on  the  simulation  settings  collectively 
agree  that  a  threat  to  the  harbour  was  imminent.  The  experimental  results  show  that  the  cognitive 
factor  has  the  greatest  impact  on  consensus  achievement,  though  social  and  informational  factors 
are  also  important. 

Significance:  The  research  to  arrive  at  the  HSE  tool  was  conducted  in  six  stages,  as  presented  in 
previous  deliverables,  and  culminates  with  a  detailed  multi-agent  simulation  involving  high- 
functioning  work-practice  agents,  following  organizational  workflows,  and  being  influenced  by 
social,  cognitive,  and  informational  factors.  Such  a  simulation  can  help  to  elicit  important 
information  about  how  these  core  factors  relate  to  one  another  and  ultimately  impact  the  system. 
This  suggests  that  continued  exploration  using  computer  simulations  is  beneficial.  Additionally, 
the  HSE  approach  provides  a  core  methodology  for  constructing  such  simulations,  and  has  the 
potential  to  be  adapted  for  future  experiments  at  relatively  low  cost. 

Future  plans:  The  lessons  learned  from  this  work  are  threefold:  (i)  more  focus  on  human  factors 
through  simulation  is  necessary  to  better  explore  the  complex  interplay  between  human  factors 
and  meta-organizational  design;  (ii)  more  focus  on  human-factor  model  validation  is  needed  to 
add  credibility  to  the  results.;  and  (iii)  more  information  on  organizational  procedures  is  needed  to 
create  meaningful  functional  models  that  can  be  “operationalized.”  These  will  lead  to  an 
improved  understanding  of  the  human  element  and  its  impact  on  joint-organizational  endeavours. 
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Modelisation  des  operations  de  securite  publique  :  Evaluation  de  la 
validation  de  principe  des  ecosystemes  holistiques  de  securite 

Alexis  Morris;  William  Ross;  Mihaela  Ulieru;  DRDC  CSS  CR  2012-026;  RDDC 
CSS  Decembre  2012 

Introduction  :  Ce  rapport  resume  et  evalue  les  recherches  effectuees  par  l’Universite  du 
Nouveau-Brunswick  (UNB)  dans  le  contexte  du  projet  de  modelisation  des  operations  de  securite 
publique  du  RDDC,  qui  vise  a  etudier  la  prise  de  decisions  dans  des  meta-organisations 
complexes.  Dans  cette  etude,  les  travaux  de  l’UNB  comportaient  la  conception  et  l’instauration 
d’une  approche  de  simulation,  la  simulation  d’ecosystemes  holistiques  de  securite  (EHS),  pour 
representer  les  organisations  et  le  processus  de  prise  de  decisions  et  pour  simuler  les 
repercussions  des  principaux  facteurs  sociaux,  cognitifs  et  informationnels  sur  l’efficacite  globale 
d’un  ecosysteme  de  securite. 

Resultats  :  La  partie  du  projet  de  RDDC  portant  sur  les  ecosystemes  holistiques  de  securite 
(EHS)  a  ete  attribuee  dans  le  but  d’examiner  les  systemes  organisationnels  conjoints  dans  une 
perspective  descendante  (individuelle)  et  ascendante  (systeme).  Elle  portait  sur  le  developpement 
d’une  approche  de  simulation  informatisee  pour  explorer  l’effet  des  principaux  facteurs  sociaux, 
cognitifs  et  informationnels  presents  chez  les  agents  individuels  (au  niveau  individuel)  sur  les 
efforts  inter  organismes  (au  niveau  du  systeme),  plus  particulierement  l’intervention  d’un 
consensus  par  echange  d’ information.  La  simulation  visait  essentiellement  a  determiner  si  les 
agents  a  la  rationalite  limitee,  temoins  seulement  de  certains  aspects  de  la  situation,  pourraient, 
conjointement  avec  d’autres  agents  et  selon  le  contexte  de  la  simulation,  s’ entendre  sur  le  fait  que 
le  port  etait  menace.  Les  resultats  experimentaux  montrent  que  c’est  le  facteur  cognitif  qui  a  la 
plus  forte  incidence  sur  l’intervention  d’un  consensus,  bien  que  les  facteurs  sociaux  et 
informationnels  soient  aussi  importants. 

Importance  :  La  recherche  qui  a  mene  a  l’outil  EHS  a  ete  menee  en  six  etapes,  comme  il  est 
explique  dans  les  documents  precedents,  et  culmine  avec  une  simulation  multi-agents  detaillee 
comportant  des  agents  de  pratiques  de  travail  de  haut  niveau,  suivant  le  deroulement  des 
operations  et  etant  influences  par  des  facteurs  sociaux,  cognitifs  et  informationnels.  Une  telle 
simulation  peut  contribuer  a  obtenir  de  l’information  importante  sur  la  fag  on  dont  ces  principaux 
facteurs  sont  lies  entre  eux  et  finissent  par  influer  sur  le  systeme,  ce  qui  laisse  penser  que  la 
poursuite  de  l’exploration  au  moyen  de  simulations  informatisees  est  utile.  De  plus,  l’approche 
EHS  procure  une  methodologie  de  base  pour  construire  de  telles  simulations  et  pourrait  etre 
adaptee  a  d’autres  experiences  a  un  cout  relativement  bas. 

Perspectives  :  Les  legons  retenues  de  ces  travaux  sont  triples  :  i)  il  est  necessaire  de  se  pencher 
davantage  sur  les  facteurs  humains  au  moyen  de  simulations  pour  mieux  explorer  les  interactions 
entre  les  facteurs  humains  et  la  conception  des  meta-organisations;  ii)  il  faut  s’employer 
davantage  a  valider  les  modeles  de  facteurs  humains  pour  accroitre  la  credibilite  des  resultats  et 
iii)  il  nous  faut  plus  d’information  sur  les  procedures  organisationnelles  pour  creer  des  modeles 
fonctionnels  valables  qui  puissent  etre  « operationnalises ».  11s  permettront  une  meilleure 
comprehension  de  l’element  humain  et  de  ses  repercussions  sur  les  entreprises  conjointes. 
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1  Introduction 


This  deliverable  summarizes  and  evaluates  research  performed  by  the  University  of  New 
Brunswick  (UNB)  in  the  context  of  the  DRDC  T1F  initiative,  "Modelling  Public  Security 
Operations ,”  which  aims  to  investigate  decision-making  in  complex  meta-organizations  [TIF 
Proposal].  The  UNB  component  of  this  initiative  has  involved  the  design  and  implementation  of  a 
simulation  approach,  the  Holistic  Security  Ecosystem  (FISE)  simulation,  for  representing 
organizations  and  the  decision-making  process  and  for  simulating  the  impact  of  key  social, 
cognitive,  and  informational  factors  on  the  joint  effectiveness  of  a  security  ecosystem1. 

The  FISE  simulation  environment  has  been  described  and  presented  along  with  methodologies  for 
organization  and  human-factor  modeling  in  Deliverables  1  to  6,  [UNB1-UNB4].  This  previous 
work  considered  a  joint-operational  harbour-security  scenario  in  which  consensus  achievement, 
based  on  information  sharing,  was  examined  across  several  dimensions.  Results  indicate  that 
cognitive  factors  have  the  greatest  impact  on  joint  meta-organizational  effectiveness,  which 
supports  the  findings  of  project  research  partners,  [Uof05]. 

The  current  deliverable  evaluates  the  HSE  simulation  in  the  light  of  existing  literature,  as  well  as 
comparing  it  to  an  in-vivo  experimental  approach.  It  also  highlights  the  strengths  and  weaknesses 
of  the  HSE  approach  using  four  criteria — utility,  feasibility,  relevance,  and  verifiability — and 
presents  lessons  learned  in  developing  the  HSE  simulation. 

1.1  Project  Milestones 

Below  are  the  primary  UNB  outcomes  for  the  TIF  research  on  the  development  of  tools, 
techniques,  and  best  practices  for  modelling  public  security  operations.  (The  current  deliverable 
is  highlighted.) 

•  Design  of  the  HSE  simulation  (Oct  2009) 

•  Implementation  of  the  HSE  simulation  (Jan  2010) 

•  Extension  of  the  HSE  simulation  to  include  social,  cognitive,  and  informational  conceptual 
models  (Sept  2010) 

•  Verification  of  the  extended  HSE  simulation  (Dec  2010) 

•  Analysis  of  the  effect  of  key  social,  cognitive,  and  informational  factors  (Apr  2011) 

•  Analysis  of  the  effect  of  security  system  relationship  configurations  on  goal  achievement 
(Aug  2011) 

•  Evaluation  of  the  HSE  "proof  of  concept"  (Feb  2012) 


1  Ecosystem  here  refers  to  a  temporary  alliance  of  organizations  working  cooperatively  toward  a  common 
objective. 
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2  Related  Literature 


The  central  concepts  involved  in  the  development  of  the  HSE  platform  relate  to  six  key 
disciplines.  These  are  (i)  action  research,  used  to  gather  early  data  on  the  processes  involved 
during  the  Harbour  Siren  (2009)  exercise  where  the  UNB  team  participated  as  academic 
observers;  (ii)  human-factor  modelling,  which  was  used  in  the  selection  of  which  human  elements 
would  be  important  to  model,  as  well  as  the  actual  model  development  and  validation;  (iii)  multi¬ 
agent  system  simulation,  which  involved  the  selection  of  tools  from  the  literature  for  designing 
and  implementing  agent  decision-making  logic,  communications,  role-relationships,  and  task- 
activities;  (iv)  organizational  modelling,  which  was  used  in  considering  the  core  policies,  norms, 
and  roles  within  organizations;  (v)  consensus  modelling,  which  focuses  on  how  consensus  is 
achieved  within  multi-agent  systems  and  was  used  after  the  UNB  research  shifted  from  response 
to  threat  detection;  and  (vi)  soft-systems  methodology  research,  which  supported  the  use  of  agent- 
based  systems,  particularly  with  respect  to  having  models  of  multiple  competing  but  valid 
perspectives  of  reality.  The  sections  below  describe  each  domain  in  more  detail,  including 
relevant  core  literature,  and  highlight  further  how  these  have  influenced  the  development  of  the 
HSE. 

2.1  Action  Research 

There  are  multiple  viewpoints  on  what  constitutes  action  research;  however,  it  may  be  said  that 
action  research  involves  developing  an  understanding  of  a  process  through  participative 
engagement  with  the  system-under-study.  This  may  be  considered  depending  on  the  system- 
under-study  as  first-person,  second-person,  or  third-person  methods,  [Marshall],  First-person 
action  research  involves  researchers’  self-inquiry  into  their  own  lives.  Second-person  action 
research  involves  joint  inquiry  into  issues  of  mutual  concern  to  two  parties,  while  third-person 
action  research  is  used  to  discuss  wide  community  systems,  like  organizations  and  inter- 
organizational  partnerships.  This  process  is  iterative,  and  relates  to  the  key  phases  of  planning 
actions,  taking  actions,  observations  of  those  actions,  and  reflecting  on  what  was  learned,  as  a 
way  to  understand  a  system,  [Altrichter].  Reflection  is  a  key  component,  and  action  research 
requires  a  group  of  representatives  in  an  organization  to  reflect  on  actions  taken  from  their  own 
experiences  and  to  evaluate  and  ultimately  manage  those  experiences,  [Altrichter].  In  more 
concrete  terms,  this  is  a  process  of  several  cyclical  steps,  as  in  [Gronhaug]:  observation, 
interpretation,  plan/action,  observation  of  outcome,  reflection,  modified  interpretation 
(explanation),  and  plan/action. 

In  terms  of  the  HSE  project,  strict  third-person  action  research  was  not  followed,  but  a  modified 
approach  emerged  in  order  to  gain  an  understanding  of  the  j  oint  organizations  involved  in  harbour 
safety  and  threat  consensus  modelling.  In  particular,  the  data  gathering  phases  of  the  project 
involved  the  participation  in  the  “Harbour  Siren”  exercise  of  2009,  where  background 
(ethnographic)  observation  of  harbour  joint-consensus  exercises  was  conducted  as  a  first  step. 
Internal  team  reflection  and  online  investigation  led  to  the  development  of  an  early  HSE  model. 
Further  developments  involved  expert  consultations  and  interviews  of  two  ex-naval  members  on 
port  processes,  which  led  to  a  refined  event  timeline  of  the  final  simulation  scenario  and  an 
understanding  of  important  human  factors.  These  were  then  discussed  with  a  multi-disciplinary 
panel  of  experts  to  gain  further  insights.  Ultimately,  this  resulted  in  the  refinement  of  the 
consensus  process  model  for  a  joint  organization  simulation  with  publish,  observe,  document,  and 
risk-check  cycle,  as  defined  in  the  previous  deliverables,  [UNB  1-4]. 
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2.2  Human  Factor  Modelling 

Human-factors  research  aims  at  understanding  the  effect  of  systems  and  their  humans-in-the-loop, 
whether  for  computer  interaction,  organizations,  or  other  complex  socio-technical  systems. 
Gaining  an  understanding  of  human  behaviours  involves  interdisciplinary  study  on  multiple 
levels,  [Vicente].  These  encompass  the  physical,  psychological,  social-team,  organizational,  and 
political  levels  of  human  behaviour,  [Vicente],  which  describe  systems  in  a  framework  for 
ergonomic  studies.  These  levels  intersect  a  vast  body  of  research.  For  instance,  third-party  action 
research  would  take  place  at  higher  levels,  such  as  the  social-team  level,  or  the  organizational  and 
political  levels.  Alternatively,  disciplines  like  cognitive  ergonomics,  [Jamieson],  and  human- 
computer-interaction  would  target  the  lower  levels:  the  physical,  psychological,  and  social  levels. 
Human-factors  research  aims  to  understand  the  human  limits  and  causal  relationships  (with  the 
environment). 

In  the  HSE  work,  the  human-factor  levels  have  been  described  according  to  a  seven-dimensional 
model,  and  have  been  used  for  the  simulation  and  research  on  modelling  social  systems  (such  as 
culture,  [Culture]).  This  seven-dimensional  framework  advocates  that  systems  be  viewed  from  the 
physical,  individual,  functional,  structural,  normative,  social,  and  informational  dimensions.  The 
human  factors  include  modelling  stress,  trust,  risk  factors,  and  cultural  factors,  as  well  as 
information  observation  and  sharing,  [UNB2,  UNB3].  System  dynamics  diagramming  is  used  to 
describe  causal  loops  among  these  factors.  System  dynamics  provides  a  vocabulary  to  describe 
the  often  fuzzy  human  factors  in  terms  of  their  key  components  and  relationships.  It  has  been 
used  to  discuss  many  social  phenomena,  particularly  for  economic  systems  and  supply-chain 
management,  [Sterman].  This  research  presents  this  use  of  system  dynamics  for  factors  such  as 
stress,  [Stress],  (where  stress  is  a  psycho-physiological  human  factor  related  to  resources  versus 
task  demands).  Risk  has  been  described  in  terms  of  a  number  of  factors  related  to  the  CAR  model 
of  competence,  authority,  and  responsibility,  [CARModel],  as  well  as  various  event-related 
probabilities  of  perceived  consequence  and  other  factors,  [UNB3]. 

2.3  Modelling  and  Simulation  with  Agents 

Multi-agent  systems  represent  a  software  design  methodology  and  a  discipline  involving 
autonomously  interacting  programs  (agents)  as  actors  in  a  collaborative  environment  (simulated 
or  otherwise).  Agent  technology  has  been  used  in  multiple  domains  to  model  and  simulate  the 
phenomena  involving  interacting  components,  [Balaji].  These  agents  may  be  classified  according 
to  their  degree  of  autonomy,  proactiveness,  reactiveness,  and  social  ability,  [Wooldridge].  Agent 
systems  also  allow  for  incorporating  models  of  the  human  factors  described  previously.  The 
physiological  properties,  such  as  aspects  of  stress,  can  be  added  to  a  model  as  well  as  the 
psychological  aspects  (e.g.,  trust)  and  the  higher  dimensions  (i.e.,  social,  organizational,  and 
political).  Social  network  theory,  influence  networks,  distributed  negotiation,  economic 
modelling,  and  game  theory  all  allow  for  exploring  behavioural  properties  within  complex  social 
systems  through  agents,  [Balaji,  Shoham].  For  example,  in  [Shoham],  agents  have  been  shown  to 
model  constraint  satisfaction,  optimization,  social  laws,  cooperative  and  non-cooperative 
behaviours,  social  choice,  and  market  theories.  Other  work  on  agent  technologies  involve 
architectures,  languages,  and  programming  frameworks,  as  described  in  [Bryson], 

In  terms  of  the  influence  of  multi-agent  system  simulation  on  the  developed  HSE  approach,  there 
are  several  core  points.  First  is  modelling  and  simulating  work  practice  with  agents,  [Brahms].  In 
this  work  a  new  architecture,  programming  language,  and  framework  were  introduced.  This 
framework  promoted  agent  design  based  on  the  concept  of  work-practice,  i.e.,  that  agents  are  not 
goal-driven,  as  is  commonly  considered  in  the  agent  community  ([Jason,  Brahms]),  but  both 
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action-  and  thought-oriented.  Brahms  agents  fit  into  a  class  of  agent  architecture  known  as  BD1, 
or  belief-desires-intentions;  an  agent  design  methodology  incorporating  a  processing  engine 
based  loosely  on  human  information  processing,  [Rao,  Wooldridge].  Such  agents  provide  a  notion 
of  bounded  rationality,  i.e.,  that  agent  knowledge  of  the  world  is  potentially  incomplete,  and 
therefore  decisions  are  made  based  on  what  beliefs  the  agent  has  of  the  world.  This  knowledge 
may  not  necessarily  map  to  the  actual  reality,  allowing  for  agent  models  that  are  not  all-knowing, 
but  instead  have  degrees  of  decision-making  abilities,  i.e.,  some  are  more  “in-touch”  with  a  world 
situation  (situationally  aware)  than  others.  These  “facts  and  beliefs”  used  in  the  agent-decision 
process  relate  to  human  factors  on  a  psychological  level.  In  Brahms,  plans  and  situated  actions 
provide  the  work  practice  of  agents  and  allow  for  inter-agent  communication  networks  to  be  set  in 
place  through  the  standard  “speech-acts”  protocol,  [Brahms]. 

Modelling  based  on  normative  behaviour  has  also  been  relevant  in  the  HSE  designs  to  model  the 
human-factor  reality  at  the  social,  organizational,  and  political  levels.  OperA  provides  a 
framework  for  modelling  rules  of  behaviour  that  map  to  actual  behavioural  norms,  [Dignum]. 
Functional  roles  and  relationships  are  also  definable  through  methodologies  like  OperA,  and  the 
HSE  project  makes  use  of  these  in  understanding  joint  organizational-roles  (described  in  the 
section  below  on  organizational  modelling). 

Finally,  the  agent  approach  taken  in  HSE  also  makes  use  of  system  dynamics  to  define  the  causal 
relationships  between  variables,  and  how  those  variables  interact  during  runtime.  This  mixed 
agent-based  and  system  dynamics  approach  has  been  discussed  in  literature  as  a  way  to  provide  a 
“best-of-both-worlds”  model  that  has  both  top-down  understanding  of  processes  (system 
dynamics)  and  bottom-up  emergent  results  from  collective  interactions  (agents).  See,  [Scholl],  for 
more  on  the  agent-based  system  dynamics  literature. 

2.4  Organizational  Modelling 

The  research  related  to  organizational  modelling  used  during  this  project  can  be  gathered  into 
three  main  groups:  enterprise  architecture,  business  process  modelling,  and  agent  organizations. 
Each  will  be  described  below. 

2.4.1  Enterprise  Architecture 

According  to  the  Enterprise  Architecture  Research  Forum,  “Enterprise  Architecture  is  the 
continuous  practice  of  describing  the  essential  elements  of  a  socio-technical  organization,  their 
relationships  to  each  other  and  to  the  environment,  in  order  to  understand  complexity  and  manage 
change,”  [EARF].  It  is  used  to  improve  the  functioning,  efficiency,  and  interoperability  of  the 
business  itself.  However,  the  important  human  element  is  missing  from  many  enterprise 
architecture  frameworks,  which  specify  the  views  of  interest  and  how  they  must  be  organized, 
[Handley].  For  example,  the  British  military  uses  MoDAF  (Ministry  of  Defence  Architectural 
Framework),  [MoDAF],  while  the  United  States  military  uses  DoDAF  (Department  of  Defence 
Architectural  Framework),  [DoDAF],  but  neither  expressly  captures  the  variety  of  important 
human  factors,  for  instance,  the  impact  of  operator  stress  on  the  system.  They  are  used  primarily 
to  help  with  acquisitions  and  to  support  the  integration  of  equipment  capabilities,  [DoDAF].  The 
former  framework  considers  the  strategic,  operational,  service-oriented,  systems,  acquisition,  and 
technical  views  of  the  military,  [MoDAF],  while  the  latter  explores  the  capability,  data  and 
information,  operational,  project,  services,  standards,  and  systems  views,  [DoDAF]. 

These  frameworks  describe  the  prescriptive  view  (i.e.,  how  things  work  ideally),  not  the 
descriptive  view  (i.e.,  how  things  work  in  reality).  Moreover,  because  the  important  human  view 
is  missing,  these  frameworks  alone  are  not  sufficient  to  analyse  the  social,  cognitive,  and 
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informational  factors  required  in  this  project.  Currently,  these  frameworks  are  also  static 
representations  that  cannot  be  “run.”  However,  recent  work  has  attempted  to  make  executable 
enterprise  architectures  for  the  purposes  of  improving  organizational  efficiency,  [Baumgarten, 
Glazner],  trying  to  model  the  behaviour  of  the  organization,  rather  than  focusing  mainly  on 
interoperability. 

2.4.2  Business  Process  Modelling 

The  term  business  process  modelling  refers  to  “all  activities  relating  to  the  transformation  of 
knowledge  about  business  systems  into  models  that  describe  the  processes  performed  by 
organizations,”  [Scholz-Reiter].  There  are  several  reasons  to  model  the  business  processes  of  an 
organization,  but  the  primary  reason  is  to  improve  organizational  efficiency.  Business  processes 
can  be  viewed  as  a  sequence  of  tasks,  and  the  order  in  which  the  tasks  are  arranged  is  a  critical 
design  variable — more  optimal  arrangements  lead  to  more  efficient  resulting  structures,  [Orman]. 
However,  these  may  have  unexpected  consequences  on  other  parts  of  the  organization,  including 
its  people,  its  strategy,  its  environment,  its  culture,  its  information  systems,  and  other  processes, 
[Giaglis] . 

Various  tools  exist  in  support  of  business  process  modelling,  including  Intalio  BPMS,  [Intalio], 
and  AR1S,  [AR1S],  and  the  standard  visual  representation  is  Business  Process  Modelling  Notation 
(BPMN),  which  unlike  UML  that  focuses  on  objects,  examines  an  organization  from  a  process 
perspective,  [Owen],  This  notation  allows  the  modeller  to  capture  the  business  processes,  decision 
points,  and  data  stored  and  exchanged  within  and  across  organizations. 

Many  tools  allow  for  the  creation  of  static  BPMN-compliant  workflows,  but  Intalio  BPMS  and 
AR1S  also  provide  the  ability  to  simulate  the  processes,  by  combining  events,  processes,  and 
rules,  [Intalio].  In  addition,  these  tools  are  also  well-suited  for  modelling  the  structure  of  the 
organization.  For  example,  AR1S  provides  a  set  of  metaclasses  to  describe  any  organization: 
Organizational  Unit,  Organization  Unit  Type,  Position,  Person,  Person  Type,  Group,  and 
Location,  [Santos].  However,  according  to,  [Glazner],  ARIS  is  limited  to  the  domain  of 
information  systems. 

This  modelling  method  is  particularly  appropriate  for  analysing  the  structural,  functional,  and 
normative  aspects  of  an  organization,  where  normative  relates  to  the  rules  specified  by  the 
organization.  However,  it  does  not  take  into  account  human  factors,  such  as  stress,  trust,  and 
influence,  nor  does  it  simulate  humans  deviating  from  the  norms.  As  such,  it  provides  a  useful 
communication  mechanism  (namely  BPMN)  and  evidence  that  parameter  tuning  can  be  used  to 
improve  organizations,  but  this  method  alone  is  not  sufficient  to  address  the  requirements  of  the 
project. 

2.4.3  Agent  Organizations 

Unlike  the  other  approaches  examined  in  this  subsection,  an  agent-based  approach  allows  the 
effect  of  individual  agent  characteristics  to  be  explored,  [Borshchev].  Whenever  two  or  more 
agents  interact  in  the  same  environment,  a  society  of  agents  exists,  and,  [Ferber],  argues  that 
elements  of  organizational  structure,  including  “groups,”  “roles,”  and  “organizations,”  can  help 
provide  the  necessary  structure  to  alleviate  some  of  the  challenges  of  classical  agent-based 
systems  (e.g.,  accurately  modelling  the  problem  being  addressed). 

A  targeted  effort  to  model  organizations  using  agents  is  found  in,  [Dignum],  OperA 
(Organizations  per  Agent)  is  a  non-executable  modelling  language  that  is  used  to  describe  the 
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interaction  between  different  agents  within  an  organization.  It  provides  constructs  to  capture  the 
structure  of  an  organization  in  terms  of  roles  and  relationships,  the  interaction  patterns  between 
agents  as  workflows,  and  the  organization’s  normative  aspects  using  deontic  logic.  Deontic  logic 
labels  an  action  as  being  obligatory >,  prohibited,  or  permissible,  but  does  not  dictate  which  action 
an  agent  must  choose.  This  flexibility  allows  agents  to  perform  actions  that  do  not  strictly  adhere 
to  an  organization’s  policies.  Because  it  is  non-executable,  OperA  was  not  sufficient  as  a  stand¬ 
alone  tool  for  the  purposes  of  this  project.  However,  for  our  first  deliverable,  [UNB1],  OperA  was 
used  in  conjunction  with  Brahms,  which  is  executable,  following  the  guidelines  outlined  in, 
[Putten]. 

2.5  Consensus  Modelling 

Group  decision  making  is  one  of  the  core  targets  for  the  HSE  system,  as  agents  in  the  simulation 
must  reach  consensus  of  a  threatening  event.  This  form  of  distributed  decision  making  has  been 
considered  in  the  literature  from  several  perspectives.  The  game-theoretic  approach  to  consensus 
modelling  highlights  some  of  the  important  properties  of  distributed  agent  simulation  as  a  whole. 
The  simulation  could  be  cooperative,  or  non-cooperative,  or  have  complete  or  incomplete 
information  for  its  agents.  Additionally,  these  agents  may  be  either  truthful  or  deceptive, 
[Shoham].  In  all  cases,  agents  act  as  utility  maximizers  for  a  specific  kind  of  utility  function. 
These  functions  may  be  designed  so  as  to  emerge  certain  properties  in  the  simulation  from  a 
collective -society  perspective.  Different  individual  utility  functions  pertain  to  different  agent 
strategies,  including  risk  perceptions  and  willingness  to  take  risky  behaviours.  These  have  been 
shown  in  various  protocols,  including  agents  forming  contracts,  making  bribes  to  other  agents, 
and  mediatorship  behaviours  between  agents,  [Shoham]. 

The  various  strategies  related  to  consensus  modelling  include  particular  voting  protocols  and  the 
types  of  core  leadership  in  place.  There  are  a  number  of  voting  protocols,  as  seen  in,  [Shoham], 
but  a  common  method  involves  the  use  of  a  simple  majority.  Other  strategies  for  decision  making 
involve  principles  of  negotiation  such  as  strategic  negotiation,  and  principles  of  alternating  offers, 
[Kraus,  Rubenstein].  Further  literature  shows  a  number  of  mathematical  approaches  to  discuss  the 
consensus  problem,  many  of  which  use  dynamic  system  modeling,  such  as,  [Fedrizzi2010, 
Campanella],  and  notions  of  opinion  dynamics,  [Blondel]. 

The  HSE  model  for  consensus  uses  a  simple-majority  rule  to  determine  the  measure  of  consensus 
regarding  the  threat-level  at  a  given  time  during  the  simulation.  This  approach  considers  a 
coalition  of  agents  for  making  the  consensus  decision,  which  is  selected  during  analysis. 
Moreover,  multiple  views  of  consensus  are  possible,  and  all  are  captured,  depending  on  the 
analyst’s  definition  of  what  percentage -threshold  constitutes  consensus,  [UNB4],  The  HSE 
approach  is  deception-free,  i.e.,  agents  are  not  allowed  to  lie  about  information  being  shared, 
making  it  (in  the  rhetoric  of  agent  literature)  a  cooperative  information-sharing  game. 

2.6  Soft  Systems  Methodology 

The  Soft  Systems  Methodology  (SSM)  is  premised  on  the  idea  that  there  is  not  one  valid 
perspective  of  “reality,”  but  rather  multiple  interacting  perspectives,  [Checklandl,  Checkland2], 
These  may  be  quite  contrary,  but  equally  valid;  for  example,  one  person’s  “information  hoarding” 
is  another’s  “protection  against  legal  action.”  These  interacting  perspectives  represent  “messy” 
problems  for  management  precisely  because  soft  factors  (e.g.,  psychology  and  culture)  tend  to 
dominate  over  “hard”  factors  (e.g.,  technical  specifications),  resulting  in  poorly-defined  problems, 
[Checklandl]. 
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Checkland’s  seven-step  SSM  offers  a  methodology  to  overcome  these  messy  problems  by 
analysing  reality  from  multiple  perspectives  and  proposing  a  change  to  the  organization  that  is 
culturally  feasible,  taking  into  account  “the  particular  history,  culture,  and  politics  of  the  people 
involved,”  [Checkland2] .  The  seven  points  in  this  methodology  are  as  follows,  [Checkland2] : 

(i)  A  perceived  problematic  situation  exists; 

(ii)  It  will  be  perceived  differently  by  people  with  different  worldviews; 

(iii)  It  will  contain  people  trying  to  act  purposefully  (intention); 

(iv)  To  understand  this  problem,  make  models  of  purposeful  activity  as  perceived  by 
different  worldviews; 

(v)  Use  models  as  a  source  of  questions  to  ask  of  the  problematical  situation,  thus 
structuring  a  discussion  about  changes  which  are  both  (a)  desirable  (given  these 
models)  and  (b)  feasible  (culturally); 

(vi)  Find  versions  of  the  to-be-changed  situation  which  different  worldviews  could 
live  with; 

(vii)  Implement  changes  to  improve  (be  ready  to  start  the  process  again). 

This  methodology  is  well-aligned  with  the  HSE  approach.  Because  it  uses  agent-based  societies, 
the  HSE  approach  is  able  to  address  points  (ii)  -  (iv)  above.  BD1  agents  have  their  own 
perspective  of  reality  which  exists  as  a  set  of  beliefs.  In  the  simulation,  agents  may  perceive 
different  indicators,  but  they  also  have  different  beliefs  regarding  the  importance  (or  potential 
risk)  of  an  indicator  (ii).  This  in  turn  will  guide  them  in  deciding  how  to  share  (or  not)  the 
indicator  with  other  agents  (iii).  With  the  ecosystem  and  organizational  agents  modelled,  the 
effect  of  various  “soft”  factors  on  the  desired  goal  of  the  system  can  then  be  simulated  (iv).  When 
a  particular  problematic  situation  is  perceived,  for  example,  information  hoarding  (i),  the  HSE 
can  facilitate  points  (v)  and  (vi)  by  enabling  the  exploration  of  different  alternatives  in  arriving  at 
a  desirable  solution  (e.g.,  increased  situational  awareness). 

2.7  Literature  Summary 

The  above  discussion  has  shown  how  diverse  these  fields  can  be  and  highlights  the  need  for  more 
interdisciplinary  research  on  human  factors  within  simulations.  While  different  tools  and 
approaches  provide  important  pieces,  none  was  sufficient  by  itself  to  explore  the  impact  of  social, 
cognitive,  and  informational  factors.  However,  the  HSE  approach  combines  several  of  these 
techniques.  For  instance,  OperA  was  used  to  represent  organizational  structures  and  contracts 
between  organizations  (deontic  logic).  BPMN  was  used  to  capture  internal  processes  within 
organizations.  This  had  the  added  feature  of  being  easily  understood  by  a  large  audience,  which 
helped  during  the  verification  and  validation  workshop.  A  multi-agent  systems  approach, 
incorporating  the  BDI  architecture  (i.e.,  Brahms),  was  used  to  represent  individual  differences  in 
perceiving  the  environment,  which  influenced  agent  responses.  As  for  the  human-factor  models, 
system  dynamics  causal  loop  diagrams  were  employed  to  capture  behaviour  and  for 
communicating  these  models  to  experts  for  validation  purposes.  More  information  on  the  HSE 
multi-dimensional  approach  is  found  in  Deliverables  1  to  6,  [UNB1-UNB4]. 
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3  Related  Approach  (University  of  Ottawa) 


The  main  focus  of  this  report  is  to  assess  the  merits  of  using  computer  simulation,  specifically  the 
HSE  approach,  to  glean  insights  into  public  security  operations  by  modelling  and  simulating 
cognitive,  social,  and  informational  factors.  However,  there  are  other  approaches  for  investigating 
such  factors,  particularly  in-vivo  simulation,  which  has  been  used  by  our  project  partner,  the 
University  of  Ottawa  (UofO).  This  section  will  compare  the  two  approaches  based  on  the  project 
work  delivered  to  date  ([Uof01-Uof05],  for  UofO  and,  [UNB1-UNB4],  for  UNB).  Specifically, 
it  will  discuss  the  two  approaches,  the  area  of  investigation  considered  by  each  group,  and  the 
experiments  carried  out.  The  main  differences  and  similarities  between  the  two  approaches  are 
summarized  in  Table  1. 


Table  1  Approach  Comparison 


University  of  Ottawa 

University  of  New  Brunswick 

Simulation  approach 

Simulation  approach 

• 

In-vivo  simulation 

• 

Agent-based  simulation 

Area  of  investigation 

Area  of  investigation 

• 

Shared  decision  making:  coordination, 

• 

Consensus  achievement  (information 

cooperation,  and  collaboration 

sharing):  cognitive,  social,  and 

• 

0  to  +1  event  timeline 

informational  factors 

Experiment 

• 

-2  to  0  event  timeline 

• 

Roles  (Participants) 

Experiment 

• 

Multiple  scenarios 

• 

Roles  (Level-0) 

• 

Human  factors  related  to  shared 

• 

Single  scenario 

decision  making  and  situational  complexity 

• 

Human  factors  related  to  cognitive, 

• 

Variables:  coordination  vs. 

social,  and  informational  factors 

collaboration;  homogenous  vs.  mixed  pods 

• 

Variables:  randomized  vs.  constant 

• 

Results:  Qualitative  analysis 

cognitive,  social,  and  informational  factors; 

• 

Results:  Quantitative  analysis 

3.1  Simulation  Approach  &  Area  of  Investigation 

Both  groups  had  a  different  mandate  to  fulfill  within  the  project.  UofO  was  tasked  with  research 
using  in-vivo  simulations  to  investigate  meta-organizational  shared  decision  making  (SDM), 
while  UNB  was  tasked  with  modelling  public  security  operations  using  computer  simulation  to 
ultimately  investigate  consensus  achievement.  For  context,  in-vivo  simulation  is  a  process  in 
which  actual  human  participants  are  observed  in  real-time  while  they  perform  activities,  and  this 
assessment,  in  addition  to  qualitative  surveys,  was  used  by  UofO  to  evaluate  different  SDM 
methods — specifically,  coordination,  cooperation,  and  collaboration — during  the  response  to 
specific  scripted  events.  Therefore,  the  event  horizon  of  interest  is  from  incident  to  response  (0  to 
+1). 

The  work  at  UNB,  based  on  the  literature  summary,  focused  on  using  a  multi-agent  simulation  to 
model,  first,  response  (early  HSE)  and,  then,  detection  (final  HSE),  in  which  specific  human 
aspects  are  simulated  and  their  effect  on  goal  achievement  analysed.  The  event  timeline  for  the 
work  ultimately  focused  on  preparation  and  detection  leading  up  to  an  incident  (-2  to  0).  In  this 
approach,  individual  agents  represent  humans,  and  their  respective  behaviour  and  beliefs  can  be 
tuned  based  on  the  discretion  of  the  modeller.  Rather  than  looking  at  specific  humans,  this 
approach  attempts  to  explore  human  behavioural  types  and  their  impact  on  the  system.  For 
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example,  what  would  happen  if  the  majority  of  the  agents  were  selfish?  Would  information  be 
shared  across  the  ecosystem?  Even  though  these  appear  different,  the  strategies  of  coordination, 
cooperation,  and  collaboration  could  certainly  impact  the  success  of  consensus  achievement,  and 
could  be  explored  in  computer  simulations.  Broadly,  SDM  is  not  limited  to  post-disaster.  It  can  be 
used  in  the  events  leading  up  to  a  disaster  to  help  achieve  better  situational  awareness  through 
information  sharing. 

3.2  Experiment  Participants 

For  its  in-vivo  experiments,  UofO  used  participants  from  the  following  organizations: 
o  Federal:  Public  Safety  and  Canadian  Forces 
o  Provincial:  Office  of  Disaster  Management,  Manitoba  Flealth 
o  Municipal:  Ottawa  Public  Flealth,  Ottawa  Fire  Department,  City  of  Ottawa 

These  participants  performed  specific  tasks  in  relation  to  a  complex  scenario  of  an  extreme  event, 
and  empirical  data  was  collected  (e.g.,  behaviour,  self-report,  and  observations)  from  the  in-vivo 
sessions.  Three  possible  scenarios  were  considered:  a  train  derailment  and  chemical 
contamination,  a  cyber-attack  and  blackout,  and  a  radiological  “dirty  bomb.” 

For  the  computer  simulations,  the  UNB  implemented  agents,  from  20  different  areas,  were  used 
and  presented  based  on  an  initial  level-0  diagram  of  harbour  organizations.  These  were  gathered 
after  an  extensive  study  of  Canadian  harbours,  a  planning  session  with  two  ex-naval  members, 
and  a  verification  and  validation  workshop  and  are  seen  below: 

o  Federal:  Public  Safety  Canada,  Canada  Customs  and  Border  Services  Agency 
(CBSA),  Transport  Canada,  Government  Operations  Centre  (GOC),  Coast 
Guard,  Marine  Communications  and  Traffic  Service  (MCTS),  Joint  Task  Force 
Atlantic  (JTFA),  Queen’s  Flarbour  Master,  Marine  Security  Operation  Centre 
(MSOC),  RCMP 

o  Provincial:  Police  of  Jurisdiction  (provincial),  Joint  Emergency  Operation  Centre 
(EMOC), 

o  Municipal:  Police  of  Jurisdiction  (municipal),  Medical  Facility 
o  Public:  Cargo  Ships,  Vessel  of  Interest,  Protest  Group,  General  Public,  Media, 
Port  Authority. 

These  agent  actors  participated  in  observing,  documenting,  risk  checking,  and  communicating 
various  problem-related  indicators  within  the  system,  as  will  be  described  further  in  the  next 
section.  Following  each  experiment,  the  impact  of  various  social,  cognitive,  and  informational 
factors  on  consensus  achievement  was  assessed.  A  single  harbour  security  scenario  was 
considered  with  two  variations:  originating  domestically  or  offshore,  each  sharing  many 
indicators  but  with  key  differences,  unique  to  the  particular  variation  of  the  threat  event. 

3.3  Human  Factors 

As  shown  in  the  Table,  both  the  UofO  and  UNB  approaches  attempt  to  examine  human  factors. 
The  ones  analysed  by  UofO  relate  to  the  model  developed  by  Femyre  et  al.,  [Uof03],  which 
explores  the  factors  that  categorize  situation  complexity  and  the  inter-organizational  approach 
used  by  the  decision-makers  (i.e.,  coordination,  cooperation,  and  collaboration)  based  on  the 
assets  and  time  available. 

DRDC  CSS  CR  2012-026 
16 


o  The  situational  complexity,  categorized  as  simple,  complicated,  or  complex,  is 
affected  by  three  key  factors:  (i)  impact,  i.e.,  scope,  severity,  timing,  media 
involvement,  and  political  process;  (ii)  uncertainty,  i.e.,  novelty,  anticipation  & 
planning,  lack  of  information  flexibility  of  frameworks,  changing  context,  and 
new  organizations  &  partners;  and  (iii)  vulnerability,  i.e.,  economic  development, 
social  capital,  competence,  and  communication 

o  Moderators  (that  can  affect  which  approach  is  used):  assets  and  time  available  to 
decision  makers 

o  Inter-organizational  approach:  coordination,  cooperation,  and  collaboration 

For  UNB,  the  human  factors  investigated  relate  to  the  three  dimensions  of  interest — social, 
cognitive,  and  informational: 

o  Social :  reputation  risk,  organizational  alignment;  and  beliefs  about  other 
organizations:  cultural  restrictiveness,  classification  clearance,  and  technical 
clearance 

o  Cognitive :  uncertainty  risk,  risk  level  threshold,  attention  overload,  context 
confusion,  task  overload,  override  policy  condition,  times  to  observe,  document, 
risk  check,  and  publish,  and  organizational  problem  scorecard  (i.e.,  indicator- 
score  mapping) 

o  Informational :  routine  and  non-routine  events,  classification  level,  ownership 
degree,  uncertainty,  accuracy,  completeness,  credibility,  and  technical 
impediment  level 

This  portion  of  the  deliverable  only  highlights  which  factors  were  considered.  Further  information 
regarding  these  factors  for  both  approaches  is  found  in  the  project-related  reports  [UNB1-UNB4, 
Uof01-Uof05], 

3.4  Experiment  Variables 

For  the  actual  experiment,  the  variables  for  the  in-vivo  simulation  (UofO)  consisted  of  the 
following: 

o  Two  conditions  related  to  problem-solving  approach  based  on  the  types  of  tasks 
(coordination  vs.  collaboration) 

o  Two  conditions  related  to  the  composition  of  pods2  according  to  organizational 
type  (homogeneous  vs.  mixed) 

o  The  approach  to  decision  making  and  inter-organization  environment  were 
influenced  by  four  pod  configurations: 

■  (i)  homogeneous  pod,  no  inter-pod  communication; 

■  (ii)  homogeneous  pod,  open  inter-pod  communication; 

■  (iii)  mixed  organizations  pod,  no  inter-pod  communication;  and 

■  (iv)  mixed  organizations  pod,  open  inter-pod  communication 

o  Dependent  variables  for  problem  solving  processes:  satisfaction  with  problem 
solving  process  (individual,  collective,  and  external  panel),  level  of  participation 
with  each  pod,  level  of  participation  between  pods,  time  spent  on  problem 


A  pod  is  composed  of  a  group  of  individuals  working  together  to  solve  a  problem. 
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solving  stages,  pattern  of  engagement  in  problem  solving  stages,  and  task  and 
group  cohesion 

o  Dependent  variables  for  problem  solving  outcomes:  decision  quality  (individual, 
collective,  and  external  panel),  satisfaction  with  problem  solving  outcome 
(individual,  collective,  and  external  panel),  level  of  agreement  on  outcome, 
changes  in  individual  and  collective  goals,  and  time  to  reach  decision 


On  the  other  hand,  for  the  agent  simulation,  the  effect  of  the  various  factors  on  consensus 
achievement  was  explored  using  these  variables: 

o  Two  conditions  for  each  of  the  three  factors-of-interest  (randomized  vs.  constant) 
o  Eight  factor  configurations  (all  combinations  of  the  three  factors): 

■  (i)  social  (constant),  cognitive  (constant),  informational  (constant); 

■  (ii)  social  (randomized),  cognitive  (constant),  informational  (constant); 

■  (iii)  social  (constant),  cognitive  (randomized),  informational  (constant); 

■  (iv)  social  (constant),  cognitive  (constant),  informational  (randomized); 

■  (v)  social  (randomized),  cognitive  (randomized),  informational 

(constant); 

■  (vi)  social  (randomized),  cognitive  (constant),  informational 

(randomized); 

■  (vii)  social  (constant),  cognitive  (randomized),  informational 

(randomized);  and 

■  (viii)  social  (randomized),  cognitive  (randomized),  informational 
(randomized) 

o  Randomization  of  key  factors — social,  cognitive,  and  informational — and  the 
ability  to  keep  certain  factors  constant  and  test  the  effect  of  another  (or 
combinations) 

o  Number  of  execution  runs  for  each  configuration  of  the  simulator:  30 


3.5  Results 

From  the  in-vivo  work  (UofO),  it  was  found  that  there  are  challenges  to  SDM  in  terms  of 
communication  and  resources  required.  Challenges  in  participating  in  a  multi-organizational 
environment  also  increase  as  the  diversity  of  the  organizations  increases:  increased  diversity  of 
opinion  leads  to  less  agreement  on  decisions  made,  and  increased  time  pressures  when  compared 
to  working  in  a  more  organizationally  homogeneous  environment.  The  data  from  the  experiment 
demonstrated  that  organizations  could  be  “instructed”  to  solve  tasks  in  either  a  collaborative  or 
coordinated  manner. 

For  the  agent  work  (UNB),  the  most  important  dimension,  i.e.,  the  one  that  had  the  greatest  effect 
on  time  to  consensus  was  the  cognitive  dimension.  This  suggests  that  modelling  the  behaviour  of 
agents  to  be  more  “empathetic”  to  another  agent/organization’s  perspective  of  an  issue  can  be 
critical.  Therefore,  the  impact  of  the  cognitive  can  also  be  improved  through  training. 

The  results  of  both  approaches  have  shown  that,  though  different  variables  and  case  studies  were 
considered,  the  effect  of  cognitive  factors  is  the  most  important.  This  is  encouraging  as  it  suggests 
that  the  biggest  roadblock  to  achieving  coordination,  cooperation,  and  collaboration  is 
psychological,  and  could  possibly  be  improved  with  training. 
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3.6  Summary 

Both  in-vivo  and  agent-based  simulations  are  supported  by  literature,  but  each  carries  its  own 
strengths  and  weaknesses.  For  example,  agent-based  simulation  allows  many  different 
configurations  to  be  tested  at  no  additional  cost  once  it  has  been  implemented,  and  factors  can  be 
constantly  modified,  depending  on  the  interface,  without  requiring  changes  to  the  underlying  code 
base.  This  has  been  accomplished  with  the  HSE.  However,  validating  the  underlying  models 
themselves  has  proven  more  challenging:  translating  social  and  psychological  theories  is  not 
always  straightforward,  and  experts  in  these  fields  often  do  not  agree  on  one  set  theory.  In  terms 
of  in-vivo  simulation,  on  the  other  hand,  one  clear  disadvantage  is  that  the  experiment  is  more 
difficult,  and  costly,  to  repeat.  However,  rich  data  can  be  extracted  from  human  participants  and 
captured  via  video  and  audio  recordings.  This  has  great  benefits  in  terms  of  testing  new  and 
existing  theories. 

Rather  than  competing  approaches,  in-vivo  and  agent-based  simulations  are  highly 
complementary,  especially  for  studies  involving  human-factors.  With  agent-based  simulations 
(and  computer  simulations  in  general),  there  is  greater  flexibility  for  testing  parameter 
configurations,  with  the  added  benefit  that  this  is  accomplished  with  little  or  no  additional  cost. 
Therefore,  this  approach  can  be  used  to  test  various  parameter  settings  and  potential  “hunches” 
before  selecting  the  setting(s)  of  most  interest  to  be  examined  in  real-life  using  in-vivo 
simulation.  This  combined  approach  would  have  the  benefit  of  then  extracting  a  more  targeted 
data-set  from  human  participants. 
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4  Evaluation  of  HSE  Proof-of-Concept 


The  introduction  highlights  the  project  milestones  and  completion  dates  for  the  UNB  component 
of  the  TIF  project.  These  have  been  packaged  into  four  deliverable  reports,  according  to  the 
schedule  shown  in  Figure  1.  This  section  provides  an  evaluation  of  the  completed  project  for  each 
report. 
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Deliverable  4: 
December  2010 


Figure  1:  Deliverable  Timeline 

Each  of  the  packages  will  be  summarized  briefly  and  considered  according  to  four  criteria: 

(i)  relevance, 

(ii)  (ii)  feasibility, 

(iii)  (iii)  utility,  and 

(iv)  verifiability. 

These  assess  the  relevance  of  each  report  to  the  overall  success  of  the  project;  the  feasibility  of 
various  elements,  e.g.,  whether  or  not  an  element  could  be  modelled;  the  utility  of  various 
elements  with  respect  to  the  project’s  main  goal,  e.g.,  did  it  help  in  understanding  the  effect  of 
key  factors;  and,  lastly,  the  verifiability  of  the  artifacts  produced.  A  summary  of  the  overall 
benefits  and  limitations  of  the  approach  is  presented  at  the  end. 

4.1  Deliverables  1  &  2 

Project  Title: 

•  Deliverable  1  —  Design  of  the  HSE  simulation  (Oct  2009) 

•  Deliverable  2  —  Implementation  of  the  HSE  simulation  (Jan  2010),  [UNB1] 

In  this  joint  deliverable  report,  summarized  in  Table  2,  the  goal  was  to  design  and  implement  a 
proof-of-concept  simulation  in  the  area  of  emergency  response.  The  decision  was  made  to  model 
the  security  scenario  after  the  Harbour  Siren  exercise  in  Halifax.  Using  two  multi-agent  tools — 
OperA  for  modelling  the  organizations  and  norms,  and  Brahms  for  executing  the  simulation — the 
initial  simulation  was  achieved  (see  Figure  2). 
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Figure  2:  Original  HSE  Modelling  &  Simulation  Process. 


As  part  of  the  early  work  on  investigating  how  to  model  an  organization,  a  five-dimensional 
approach  was  created  which  considered  the  structural,  functional,  normative,  human,  and  physical 
dimensions.  In  this  early  simulator,  the  structural  and  normative  factors  were  examined,  and  a 
visualization  tool  created. 


Table  2:  Deliverables  1  &  2  Summary. 


Factor 

Deliverable  Highlights 

Objective 

•  Develop  "proof-of-concept"  simulation 

Case  Study 

•  Harbour  Siren  exercise 

Timeline 

•  0  to  +2  (response  and  recovery) 

M  &  S  Tools 

•  OperA  (Agent  modelling  framework) 

•  Brahms  (Agent  simulation  environment) 

o  Workshop  with  Brahms  creator 

HSE 

Simulation 
(version  1) 

•  5-D  approach 

•  Four  rule  sets  (i.e.,  configurations) 

o  Structural  and  normative  factors 

•  Focus  on  response 

o  Organizations 

■  Many  agents  per  organization 
o  Processes 

■  Basic 

■  Not  validated 
o  World  events 

■  Fire  on  a  ship 

■  Spread  of  contaminant 
o  Severity  of  disaster 

■  Escalation  as  figure -of-merit 

Visualization 

•  G1S  map  of  harbour 

•  Show  response  actions  and  communications 

•  Timeline  of  agent  actions 
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4.1.1  Evaluation 


The  proof-of-concept  simulator  development  was  essential  as  the  starting  point  for  the  UNB 
portion  of  the  project  and  ultimately  led  to  the  refinement  of  a  central  methodology  to  conduct 
policy-oriented  agent-based  simulation.  It  proved  the  feasibility  and  usefulness  of  having  such 
simulations  on  a  small-scale,  but  did  not  validate  the  assumptions  made  during  the  initial  design 
and  implementation  stages.  A  consideration  of  the  evaluation  criteria  appears  below. 

Relevance 

•  Created  a  simulator  to  test  different  configurations 

•  Explored  different  normative  and  structural  factors 

Feasibility 

•  Possible  to  model  normative  behaviours  using  OperA 

•  Possible  to  simulate  work  practice  in  multi-agent  system  using  Brahms 

•  Not  possible  to  collect  real-world  emergency  response  and  defence  workflows  due  to 
sensitivity  of  information 

Utility 

•  Early  HSE  useful  as  a  proof-of-concept;  it  showed  the  importance  of  the  structural  factor 
during  emergency  response 

•  The  rule  sets  (i.e.,  configurations)  provided  a  template  for  conducting  experiments  that 
was  used  in  the  final  version  of  the  HSE  simulation 

Verifiability 

•  Tool  can  compare  different  configurations  and  results  match  what  is  expected  in  reality 

•  Response  actions  and  action-to-organization  mappings  were  not  validated 

4.2  Deliverable  3 

Project  Title: 

•  Extension  of  the  HSE  simulation  to  include  social,  cognitive,  and  informational  conceptual 
models  (Sept  2010),  [UNB2] 

In  this  deliverable,  summarized  in  Table  3,  work  towards  extending  the  earlier  HSE  approach  was 
presented.  The  main  change  was  a  shift  in  focus  from  response  and  recovery  to  threat  detection, 
due  to  limited  access  to  real-world  information  on  the  (often)  sensitive  processes  related  to 
response  and  recovery. 

From  this  deliverable,  the  major  artifacts  included  a  new,  extensible  resource  library  (ERL3) 
framework  (see  Figure  3),  showing  how  general  development  classes  (e.g.,  Unit,  Role, 
HumanFactor  Model,  and  Organization)  are  linked  and  how  they  can  be  instantiated  with  various 
object  instances  (e.g.,  new  organizations  and  human  factors).  This  was  critical  to  allow  for  more 
general-purpose  simulation  models,  rather  than  scenario-specific  models  like  the  earlier 
simulation. 


3  Formerly,  the  emergency-response  library 
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Figure  3:  Early  ERL  Framework. 

Human  factor  models  were  also  developed  using  system  dynamics,  both  to  capture  the  feedback 
mechanisms  and  as  a  tool  to  communicate  the  model  to  experts,  [Stress].  Moreover,  work  on  the 
new  threat-detection  scenario  was  also  begun,  including  developing  and  populating  a  problem- 
indicator  matrix  (the  indicators  related  to  specific  types  of  problems)  and  an  indicator-risk  matrix 
(the  risk  level  associated  with  a  particular  indicator).  These  laid  the  groundwork  for  future 
deliverables. 


Table  3:  Deliverable  3  Summary. 


Factor 

Deliverable  Highlights 

Objective 

• 

Extending  the  HSE  approach  (from  Deliverables  1  &  2) 

Case  Study 

• 

Harbour  security 

Timeline 

• 

-2  to  0  (preparation  and  detection) 

Modelling 

• 

5-D  approach 

Methodology 

• 

ERL  (generic) 

o  Problem  tier 
o  Organization  tier 
o  Physical  tier 

• 

Human  factors 

o  System  dynamics 
o  Stress  model 
o  Culture  model 

Simulation 

• 

Stress 

o  System  dynamics  model 

Scenario 

• 

Problem-indicator  matrix 

Development 

• 

Indicator-risk  matrix 

4.2.1  Evaluation 

This  deliverable  resulted  in  key  artifacts  and  methods,  particularly  the  general  ERL  modelling 
component  and  the  shift  towards  threat  consensus.  These  insights  led  to  a  better  understanding  of 
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how  to  model  fuzzy  human-factor  elements  using  system  dynamics.  However,  it  also  highlighted 
the  need  to  validate  these  fuzzy  models.  A  consideration  of  the  evaluation  criteria  appears  below. 

Relevance 

•  Extended  the  approach  to  adapt  to  multiple  cases 

•  Incorporated  all  five  dimensions  into  the  simulation 

•  Shifted  the  focus  from  detection  to  consensus  achievement 

•  Emphasis  on  soft  modelling 

Feasibility 

•  Possible  to  create  a  general  framework  using  software  engineering  practices  (URL-based) 

•  Possible  to  create  and  communicate  soft  models  to  experts  using  system  dynamics 

Utility 

•  Useful  to  have  a  single  interface  for  multiple  cases 

•  Stress  model  was  useful  as  a  proof-of-concept  for  the  benefits  of  system  dynamics  in  soft 
modelling 

Verifiability 

•  The  stress  model  helped  highlight  not  only  the  need  for  soft  model  validation,  but  also  the 
challenges  it  presents;  for  example,  how  the  various  factors  of  stress  interrelate  is  not 
agreed  upon  by  experts 

4.3  Deliverable  4 

Project  Title: 

•  Verification  of  the  extended  HSE  simulation  (Dec  2010),  [UNB3] 

This  deliverable  centered  on  the  verification  and  validation  workshop.  The  scenario  which  was 
begun  in  the  previous  deliverable  was  extended  by  interviewing  ex-naval  members  and  included 
two  variations:  a  domestic  attack  and  an  offshore  attack.  The  scenario  artifacts  include  event 
indicators,  timelines,  impediments  to  information  sharing  and  indicator  observation,  and  a  level-0 
diagram  highlighting  the  ecosystem  organizations  of  interest. 

The  mental  process  undertaken  by  each  organizational  agent  was  also  developed  at  this  stage.  The 
four  key  processes  that  each  organizational  agent  follows  are  as  follows:  observe  (perceive  an 
indicator  either  first-hand  or  via  communication),  document  (record  the  fact  that  the  indicator  has 
been  observed),  risk  check  (test  the  indicator  against  a  list  of  possible  problems;  list  is  unique  to 
each  organization),  and  publish  (the  potential  to  publish,  i.e.,  share,  the  information  with  other 
organizations  depending  on  need-to-know  or  need-to-share  requirements). 
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Figure  4:  Early  Risk-Assessment  Model. 


At  the  workshop,  these  latest  artifacts,  including  the  risk-assessment  model  (see  Figure  4),  and 
selected  artifacts  from  the  previous  deliverables  were  presented  and  scrutinized  by  experts  from 
different  fields  and  organizations  as  summarized  in  Table  4. 


Table  4:  Deliverable  4  Summary. 


Factor 

Deliverable  Highlights 

Objective 

•  Model  extension 

•  Scenario  extension 

•  Verification  &  Validation 

Case  Study 

•  Harbour  security 

o  Domestic  attack 
o  Offshore  attack 

Timeline 

•  -2  to  0  (preparation  and  detection) 

Scenario 

Development 

•  Expert  consultation 

o  Two  ex-naval  members 

•  Two  scenarios 

o  Domestic  attack 
o  Offshore  attack 

•  Artifacts 

o  Event  indicators 
o  Timeline 

o  Observation  impediments 
o  Information-sharing  impediments 
o  Level-0  diagram 

Simulation 

Design 

•  System  architecture  diagram 

•  Simulation  process  models  (BPMN) 

o  Observe 
o  Document 
o  Risk  check 
o  Publish 
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Verification 

•  Artifacts  evaluated 

&  Validation 

o 

5-D  approach 

Workshop 

o 

Modelling  methodology 

o 

Eluman-factor  modelling  methodology 

o 

ERL  architecture 

o 

Human-factor  taxonomy 

o 

Stress  model 

o 

Risk  assessment  model 

o 

Information  sharing  impediments 

o 

Observation  impediments 

o 

Revised  problem  statement 

o 

Level-0  diagram 

o 

Publish-subscribe  model 

o 

Simulation  process  models 

•  Experts 

o 

Canadian  Coast  Guard 

o 

Marine  Security  Operations  Center  (MSOC) 

o 

Navy 

o 

Firefighters 

o 

Academia 

o 

Defence  Scientists  (RCMP) 

o 

Defence  Scientists  (M&S) 

o 

Defence  Scientists  (MDA) 

o 

Industry 

•  Survey 

4.3.1  Evaluation 

The  extended  simulation  capability  was  validated  in  a  two-phase  manner  involving  expert 
interviews  and  a  verification  and  validation  workshop  with  participants  from  multiple 
organizations.  This  deliverable  provided  the  core  situation  being  simulated  and  allowed  for 
eliciting  feedback  on  the  approaches  and  assumptions  made  during  the  design  process.  A 
consideration  of  the  evaluation  criteria  appears  below. 

Relevance 

•  Selected  the  key  factors  to  be  included  in  the  next  version  of  the  HSE  simulation 

•  Developed  a  scenario  to  test  key  cognitive,  social,  and  informational  factors 

Feasibility 

•  Two  plausible  scenarios,  with  variations,  were  designed  with  the  help  of  experts 

•  Continued  using  system  dynamics  for  soft  models 

•  A  workshop  with  experts  from  various  organizations  was  conducted  as  highlighted  in 
Table  3 

Utility 

•  Models  were  scrutinized  by  workshop  members  and  feedback  was  provided 

•  Reaction  from  the  community  was  positive;  however,  there  was  scepticism  about  whether 
such  a  simulation  was  possible 


DRDC  CSS  CR  2012-026 

26 


Verifiability 

•  The  models  and  artifacts  validated  at  the  workshop  are  summarized  in  Table  3 


4.4  Deliverables  5  &  6 

Project  Title: 

•  Deliverable  5  —  Analysis  of  the  effect  of  key  social,  cognitive,  and  informational  factors  (Apr 
2011) 

•  Deliverable  6  —  Analysis  of  the  effect  of  security  system  relationship  configurations  on  goal 
achievement  (Aug  2011),  [UNB4] 

For  the  final  deliverables,  summarized  in  Table  5,  the  models  presented  at  the  workshop  were 
refined  based  on  written  and  verbal  feedback.  The  biggest  change  was  the  refinement  of  the  five¬ 
dimensional  model  to  include  seven  dimensions:  structural,  functional,  normative,  social 
(formerly  part  of  human),  individual  (formerly  part  of  human),  physical,  and  informational. 
Moreover,  based  on  feedback,  various  properties  were  added  to  the  informational  dimension  and 
served  as  factors  that  helped  determine  whether  or  not  an  agent  would  share/publish  information 
about  the  indicator  to  other  organizations. 


Figure  5:  HSE  Simulation  Interface  and  Sample  Output. 


For  Deliverable  5,  the  new  simulation  was  implemented  and  presented  (see  Figure  5).  It  includes 
a  spreadsheet  front-end  for  selecting  which  models  from  the  ERL  to  include  in  the  simulation.  It 
also  provides  an  experiment  control  panel,  where  the  various  social,  cognitive,  and  informational 
properties  and  factors  can  be  turned  on  or  off  and  also  randomized. 
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For  Deliverable  6,  a  set  of  eight  experiments  were  run  with  specific  combinations  of  factors,  and 
the  results  were  analyzed.  It  was  found  that  cognitive  factors  had  the  greatest  impact  on 
consensus  achievement  in  the  simulation.  Also,  various  visualization  elements  were  able  to  be 
captured  (i.e.,  Cl  to  C4),  allowing  for  the  analysis  of  system-level  properties. 


Table  5:  Deliverables  5  &  6  Summary. 


Factor 

Deliverable  Highlights 

Objective 

•  Refine  models  based  on  V&V  feedback 

•  Simulation  implementation 

•  Simulation  experiment  design  &  results 

Case  Study 

•  Harbour  security 

o  Domestic  attack 
o  Offshore  attack 

Timeline 

•  -2  to  0  (preparation  and  detection) 

Models 

Refined 

•  Expanded  5-D  to  7-D 

•  Added  information  properties 

o  Ownership  degree 
o  Uncertainty 
o  Accuracy 
o  Completeness 
o  Credibility 

Simulation 

•  Architectural  components 

o  Excel  interface 
o  Brahms  code 

•  Output  (4  charts) 

o  Cl:  Perceived  threat  levels 
o  C2:  Sharing  events 
o  C3:  Observation  impediments 
o  C4:  Publishing  impediments 

Experiments 

•  Parameter  setting 
o  Social 
o  Cognitive 
o  Informational 

4.4.1  Evaluation 

The  final  deliverables  incorporated  the  validated  elements  from  the  expert  meetings  into  a 
working  simulation  of  inter-organizational  response.  Deliverable  6  tested  all  possible 
combinations  of  cognitive,  social,  and  informational  factors  within  the  simulation.  This  achieved 
the  goals  of  the  project,  with  results  that  were  reasonable  and  based  on  large  averages  over  a 
significant  number  of  simulation  runs.  The  HSE  approach  showed  that  agent-based  programming 
is  feasible  for  human-factor  modelling.  A  consideration  of  the  evaluation  criteria  appears  below. 

Relevance 

•  Implemented  the  models  presented  at  the  verification  and  validation  workshop 
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•  Tested  the  effect  of  various  elements  on  the  three  key  factors  of  interest:  cognitive,  social, 
and  informational 

Feasibility 

•  Tools  had  already  been  tested  and  developed  (i.e.,  Brahms  and  ERL)  and  a  familiar 
interface  available  (i.e.,  Excel) 

Utility 

•  Platform  to  test  the  effects  of  various  elements  on  the  three  key  factors 
Verifiability 

•  Individual  models  were  validated  during  the  workshop;  however,  their  interactions  were 
not  validated 

•  Following  the  implementation,  the  simulation  software  was  not  exhaustively  tested  (e.g., 
black-box  testing) 

4.5  Overall  Evaluation 

The  HSE  simulation  environment  has  accomplished  the  project  goals.  It  simulated  inter- 
organizational  communication  showing  the  impact  of  information  observation  and  sharing  on 
organizational  risk  assessment.  Additionally,  the  human  element  was  able  to  be  incorporated, 
which  is  useful  for  future  research  efforts.  The  refined  seven-dimensional  model  is  novel  and 
provides  a  methodology  for  exploring  the  complexities  of  organizations.  The  system  dynamics 
approach  to  understanding  causal  relationships  between  human  factors  proved  to  be  effective  and 
useful  in  validation  efforts.  The  implementation  of  the  ERL  based  on  the  seven-dimensional 
model,  business  processes  for  information  sharing,  and  human-factor  models  has  shown  to 
provide  a  basis  for  further  agent  simulation  research  in  this  area.  Lastly,  it  provides  a  repeatable, 
computable  tool  for  exploring  and  understanding  human-factor  effects  through  agent-based 
simulation. 

The  core  benefits  and  limitations  of  the  LISE  approach  are  shown  below: 

4.5.1  Benefits 

•  General  approach 

o  Allows  for  the  inclusion  of  new  models  and  scenarios 

o  Respects  the  complexity  of  reality  through  a  seven-dimensional  lens  used  to 
create  interrelated  models 

•  Spreadsheet  interface 

o  Allows  for  parameter  variation  and  model  selection 
o  Elides  underlying  coding  details  from  the  user 

4.5.2  Limitations 

•  Difficult  to  verify  the  interaction  of  multiple  human-factor  models 

o  No  agreed  model  for  many  of  the  human  factors  being  investigated 
o  No  clear  direction  for  validating  the  interplay  of  several  interacting  models 

•  Difficult  for  university  researchers  to  get  access  to  real-world  data,  such  as  actual 
business  practices 


29 


5  Lessons  Learned 


The  general  reflection  of  the  work  is  that  the  HSE-portion  of  the  project  has  been  successfully 
accomplished,  especially  with  regards  to  the  work  on  approaches  and  methods  for  building 
simulations  that  incorporate  human-factor  perspectives.  This  work,  even  as  a  proof-of-concept, 
may  prove  informative  to  researchers  and  theorists  looking  to  experiment  with  best-practice 
discovery  and  policy  analysis.  A  number  of  important  challenges  were  encountered  over  the 
course  of  the  project,  and  these  have  been  summarized  as  lessons  learned  in  this  section,  and  are 
presented,  as  in  the  evaluation  section,  by  report  deliverable. 

The  lessons  learned  cover  all  phases  of  the  project,  from  preliminary  designs  through  to  final 
outcomes,  and  aim  to  provide  insights,  particularly  for  future  studies  on  incorporating  human 
factors  into  simulations.  The  core  deliverables  have  been  explained  in  the  section  above  and 
cover  (i)  the  development  of  a  proof-of-concept  HSE  simulation,  (ii)  the  extension  of  the  EISE 
simulation  approach,  (iii)  the  extension,  verification,  and  validation  of  EISE  models  and  scenarios, 
and  finally  (iv)  the  refinement  of  models  and  analysis  of  key  factors  using  simulation 
experiments. 

The  following  subsections  describe  the  issues,  outcomes,  and  lessons  learned  from  each 
deliverable  (or  combined  deliverables). 


5.1  Deliverables  1  &  2 

Project  Title: 

•  Deliverable  1  —  Design  of  the  EISE  simulation  (Oct  2009) 

•  Deliverable  2  —  Implementation  of  the  EISE  simulation  (Jan  2010),  [UNB1] 

The  early  deliverables  aimed  to  develop  an  initial  simulation  as  a  proof-of-concept,  representing 
the  plausibility  of  using  simulations  to  understand  the  impact  of  various  decisions  and  policies  on 
an  unfolding  incident.  The  Elarbour  Siren  (2009)  exercise  provided  the  backdrop  scenario  for  the 
simulation,  including  key  policies  to  explore.  A  small  agent-oriented  meta-organization  was 
developed  and  a  “what-if’  analysis  undertaken.  The  core  challenge  during  this  phase  of  the 
project  was  on  the  establishment  of  the  initial  methodology,  and  key  issues  and  outcomes  are 
shown  in  Table  6. 


Table  6:  Developing  an  Initial  Methodology  (Deliverables  1  &  2). 


No. 

Issues 

Outcomes 

1 

What  are  the  important 
variables? 

Five  Dimensional  Modelling  (5-D) 

•  Organizational  structure 
parameters 

•  Normative  parameters 

•  Functional  parameters 

•  Human-factor  parameters 

•  Physical  parameters 

2 

Availability  of  information 
•  How  to  validate 

organizational  process 

Elarbour  Siren  exercise 
•  Significant  progress  was  made 
following  visit  to  Harbour 
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models? 

Siren  exercise 

3 

Model  details  overwhelming 
•  What  level  of  detail  to 

use? 

Focused  on  important  processes, 
not  objects  (e.g.,  telephone 
system) 

4 

What  tools  to  use? 

Learned  about  important  tools 

•  Brahms 

•  OperA 

5 

What  to  measure?  How  to 
show  results? 

Rule  sets  (deontic  logic)  /  charts 

The  first  issue  addressed  the  need  to  discover  the  key  variables  for  the  initial  HSE  approach.  The 
primary  outcome  is  the  result  of  an  investigation  of  modelling  parameters  from  a  multi¬ 
dimensional  perspective.  In  particular,  this  led  to  the  insight  that  at  least  five  kinds  of  parameters 
would  be  essential  for  simulating  organizations:  (i)  the  organizational  structure;  (ii)  the  normative 
rules  within  that  organization  that  govern  the  behaviour  of  its  actors  (i.e.,  organizational 
members);  (iii)  the  functional  role-based  capacities  of  key  actors;  (iv)  the  elusive,  yet  important, 
“human-factor”  elements;  and  (v)  the  everyday  physical  environmental  objects  and  parameters. 
This  “lens”  has  been  used  throughout  the  project,  with  some  additions,  and  was  particularly 
useful  as  a  starting  point  for  describing  organizations. 

The  second  challenge  involved  obtaining  valid  information  from  stakeholders  to  set  core 
parameters.  As  there  are  security  and  cultural  obstacles  to  information  sharing  for  academic  use, 
the  solution  for  obtaining  organizational  data  was  to  use  a  semi-ethnographic  approach: 
participate  in  the  Harbour  Siren  exercise  (2009)  as  external  academic  observers.  This  provided 
core  information  for  the  scenario  and  valuable  information  through  observation.  This  type  of 
approach  is  considered  appropriate,  although  very  opportunistic.  It  would  be  better  not  to  have 
needed  to  rely  on  participation  in  this  exercise  to  secure  the  necessary  modelling  data. 

The  third  issue  addressed  the  level  of  detail  to  include  in  the  simulation.  At  this  stage,  the 
simulation  of  meta-organization  acting  in  a  crisis  situation  represented  a  host  of  possible 
simulation  perspectives,  and  the  core  dilemma  of  which  elements  of  the  5-dimensional  model  to 
focus  on  presented  a  significant  decision  point.  The  result  was  to  scope  the  work  towards 
processes,  rather  than  simulation  of  objects  themselves.  This  downplayed  the  physical  simulation 
dimension,  while  augmenting  the  functional  and  normative  dimensions.  As  a  result,  it  was 
possible  to  focus  on  select  analyses  of  decision-making  and  what-if  situations.  This  proved  useful, 
as  it  translated  directly  to  the  requirements  of  the  project  and  removed  focus  on  simulation 
physics,  which  have  many  parameters  that  distract  from  the  true  purpose  of  the  simulation. 

The  fourth  issue  involved  the  selection  of  appropriate  modelling  and  simulation  tools  and 
technologies  for  the  development  of  the  HSE  simulation.  This  refers  primarily  to  the  selection  of 
an  agent-oriented  methodology  for  organizational  modelling  (OperA,  [Dignum])  and  simulation 
(Brahms,  [Clanceyl998]).  OperA  represents  an  easy  to  understand  method  of  developing 
simulation  parameters  that  specially  concentrate  on  norms,  structures,  and  role  functions.  Brahms 
provides  an  agent  engine  based  on  recent  advances  in  multi-agent  systems  and  A1  research, 
especially  the  concept  of  work-practices,  or  what  people  “actually”  do,  rather  than  what  their 
goals  are.  These  tools,  when  combined,  represent  a  very  considerable  technology  for  exploring 
real-world  organizations  through  computer  simulation. 

The  fifth  issue  involved  the  decision  of  which  outcome  metrics  to  derive  from  the  simulation  and 
how  best  to  present  them.  It  was  decided  to  perform  an  assessment  based  on  deontic  logic  (i.e., 
permission,  obligation,  and  prohibition)  of  the  normative  rule/policy  decisions  of  agents,  as  well 
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as  the  structural  makeup  of  agent  organizations.  These  allowed  for  the  simulation  of  specific 
what-if  scenarios  and  resulted  in  targeting  the  “cause-and-effect”  of  the  simulation  results,  which 
were  presented  as  graphs  using  metrics  such  as  time-to-solve  problem,  amount  of  damage, 
resources  used/unused,  and  also  problem  escalation  level.  This  outcome  was  useful  in  discussing 
the  simulation  with  stakeholders. 

5.1.1  General  Comments 

In  general,  this  phase  of  the  simulation  provided  the  proof  that  simulations  of  human 
organizations  are  useful  in  conducting  what-if  analysis  of  important  decision  factors. 
Additionally,  the  human  factors  incorporated  into  the  simulation  at  this  point  were  facilitated  by 
the  use  of  cognitive  agent  architecture,  Brahms.  The  simulation  and  methodology  had  merit,  but 
the  lack  of  real-world  data  resulted  in  a  change  of  scenario  from  response  to  prevention  in  June 
2010. 


5.2  Deliverable  3 

Project  Title: 

•  Extension  of  the  HSE  simulation  to  include  social,  cognitive,  and  informational  conceptual 
models  (Sept  2010),  [UNB2] 

Elaving  an  initial  proof-of-concept  provided  the  foundation  for  a  base  framework  for  the  EISE 
approach.  The  next  step  involved  the  extension  of  the  HSE  approach,  from  a  modelling 
perspective.  The  primary  concerns  at  this  point  involved  making  the  approach  generic  and 
expanding  the  human  element  in  the  approach,  as  highlighted  in  Table  7. 


Table  7:  Extending  the  HSE  Approach  (Deliverable  3). 


No. 

Issues 

Outcomes 

1 

•  How  to  make  process 
more  general/generic? 

•  ERL  (reusable  models) 

2 

•  Elements  missing  from 
5D  approach? 

o  Technology 
o  Information 

•  Seven-Dimensional  Modelling 
(7-D)  (Deliverables  5  &  6) 
o  Information  added  to 
model 

o  Technology  part  of 
physical 

o  Human  separated  into 
individual  (including 
psychological)  and 
social 

3 

•  How  to  capture  human 
elements? 

•  Exploration  of  system 
dynamics 

o  Stress  model 
o  High-level  human- 
factor  models 

4 

•  How  to  overcome  lack 
of  infonnation? 

•  Change  scenario  from 
response  to  prevention 
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o  Scenario  development 
o  Added  expert 
information  helps 
scenario 

o  Read  literature  on 
Harbour  procedures 


The  first  issue  involved  a  study  of  possibilities  for  generalizing  the  original  simulation,  allowing 
the  approach  to  be  applicable  to  future  scenarios.  The  outcome  was  a  refined  process  and  a 
decision  to  focus  on  a  general-purpose  simulation  library:  the  Extensible-Resource  Library  (ERL) 
for  emergency-response  simulations.  This  library  guided  a  redesign  of  the  core  simulation 
components,  having  in  view  a  wider  perspective.  The  key  lesson  is  the  need  to  address  generality 
in  simulations  early  in  the  design  process. 

The  second  issue  involved  the  expansion  of  the  five-dimensional  approach  to  incoiporate  two 
new  factors.  The  properties  of  technology  and  information  were  both  unaccounted  for,  but  were 
considered  particularly  relevant  factors.  As  a  result,  the  seven-dimensional  modelling 
methodology  was  created,  adding  information  as  its  own  dimension  and  incorporating  technology 
as  part  of  the  physical  dimension.  Additionally,  the  human  perspective  was  separated  into 
individual  properties  (including  psychological)  and  social  properties.  The  core  lesson  involves  the 
creation  of  a  holistic  lens  for  modelling  organizations.  It  is  recommended  that  future  works  adopt 
the  seven-dimensional  approach. 

The  third  issue  involved  the  methods  to  describe  and  capture  the  fuzzy  human  models  related  to 
the  social,  cognitive,  and  informational  factors  needed  in  the  simulation.  This  led  to  an 
exploration  of  specific  model  parameters  and  a  selection  of  an  approach  to  understand  the  overall 
impact  of  these  parameters  on  the  model.  The  system  dynamics  approach  was  applied  to 
modelling  initial  cognitive  factors,  namely  stress  [Stress],  in  a  fashion  that  allows  for  validation 
by  experts.  It  was  discovered  that  the  approach  indeed  was  useful  to  incorporate  high-level  human 
factors,  enabling  the  extraction  of  core  components,  as  well  as  the  relationships  between  them, 
and  allowing  for  the  eventual  inclusion  of  cognitive-factor  dynamics  into  the  simulation  at 
runtime. 

The  fourth  issue  involved  addressing  the  lack  of  information  for  a  general-purpose  scenario, 
loosely  related  to  the  earlier  one.  The  outcome  was  to  switch  the  target  focus  for  the  simulation 
away  from  complicated  response-activity  modelling,  which  requires  detailed  business  practices 
that  may  be  proprietary  or  security-sensitive.  The  new  focus  involved  looking  at  the  issue  of 
prevention,  particularly  achieving  consensus  across  the  ecosystem  about  whether  or  not  a 
particular  security  threat  was  imminent.  This  still  involved  gaining  access  to  expert  information 
through  third-party  discussions  and  publicly  available  literature  on  harbour  procedures. 
Combined,  these  proved  to  be  useful  for  information  extraction  and  scenario  development. 

5.2.1  General  Comments 

The  central  challenge  of  the  third  deliverable  was  to  enhance  the  HSE  approach.  The  general- 
purpose  goal  of  the  system  designs  and  models  was  carried  through  to  the  remaining  deliverables 
and  full-testing  of  the  generality  remains  to  be  explored  in  future  scenarios.  The  need  for 
generality,  however,  brings  with  it  an  added  overhead  that  requires  early  simulation  efforts  to  be 
reworked  in  order  to  fit  into  the  new  framework. 
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5.3  Deliverable  4 


Project  Title: 

•  Verification  of  the  extended  HSE  simulation  (Dec  2010),  [UNB3] 

Having  refined  the  HSE  approach  in  the  previous  deliverable,  the  subsequent  phase  involved 
verification  and  validation  of  the  approach  and  associated  models.  The  decision  was  to  conduct  an 
expert  validation.  This  resulted  in  the  2010  roundtable  discussion  and  presentations,  attended  by 
experts  from  governmental  agencies,  industry,  and  academia.  Useful  recommendations  were 
brought  forward  during  the  workshop,  and  the  core  challenges  in  conducting  the  session  are  seen 
in  Table  8. 


Table  8:  Verification  and  Validation  of  the  HSE  Approach  (Deliverable  4). 


No. 

Issues 

Outcomes 

1 

•  How  to  gather 

information  from  expert 
group? 

•  Survey  was  conducted 
based  on  meeting 

•  Iterative 
validation 
would  have 
been  useful 

•  Need  for  further 
validations, 
such  as 
presenting 
responses  to 
feedback  of  the 
expert  group 

•  Surveys  were  useful  in 
gathering  feedback 

2 

•  Selecting  a  core  expert 
group 

•  Volunteers 

o  Mixed  group 

3 

•  How  to  structure 

information  for  expert 
validation? 

o  Stakeholder 
customized 
data  views 

•  Used  established 

methods  of  information 
display 

o  System 

Dynamics 
o  Business 

Process 

Modelling 

Notation 

4 

•  Validating  simulation 
was  not  achievable 
(initial  goal) 

o  Timelines 
o  New  scenario 

•  Validation  of  models 
prior  to  implementation 

•  More  psychological 
input  needed  from 
experts 
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The  first  issue  centered  on  the  information  gathering  and  knowledge  acquisition  problem.  It  was 
essential  to  decide  which  method  to  use  to  elicit  feedback  from  the  participants  in  the  expert 
group.  Ultimately,  it  was  decided  that  a  survey,  in  combination  with  the  live  roundtable 
discussion,  would  be  used.  The  assessment  of  this  combined  approach  is  that  it  allows  for 
targeted,  yet  non-restrictive,  verbal  feedback  during  the  session  (e.g.,  the  survey  can  guide 
discussions),  and  also  provides  a  useful  follow-up  mechanism  for  recording  private  reservations 
and  recommendations.  However,  it  would  have  been  useful  to  conduct  further  sessions,  so  that  the 
outcomes  at  later  stages  of  the  project  could  be  iteratively  validated. 

The  second  issue  involved  the  problem  of  deciding  which  experts  should  be  included,  and  gaining 
access  to  these  experts  for  the  workshop.  In  the  end,  a  mixed  group  of  volunteers  from  various 
organizations  were  in  attendance;  however,  it  is  noted  that  there  were  relatively  few  participants 
from  the  psychological  community,  an  important  expert  class. 

The  third  challenge  involved  information  presentation  and  dissemination  to  the  expert  group,  and 
particularly  how  to  structure  this  information  for  stakeholders  with  varying  viewpoints.  The 
outcome  was  to  focus  on  using  well-established  information  display  mechanisms,  specifically 
system  dynamics  causal  loop  diagrams  and  the  business  process  modelling  notation  (BPMN). 
These  were  found  to  be  very  adequate  for  describing  concepts  and  workflows. 

The  fourth  challenge  involved  validating  the  simulation  at  the  workshop,  as  the  new  simulation 
remained  to  be  implemented.  The  event  timelines  and  general  scenario  were  presented,  but  were 
not  discussed  at  length.  Instead,  the  human-factor  models  were  the  central  focus,  and  only  a  short 
discussion  of  the  scenario-specific  artifacts  was  conducted.  This  allowed  for  the  maximization  of 
the  pool  of  experts,  as  scenario-specific  artifacts  can  easily  change  within  and  across  scenarios 
and  are  not  as  general  as  the  human-factor  models. 

5.3.1  General  Comments 

In  general,  the  verification  and  validation  went  well  allowing  the  simulation  implementation  to 
proceed.  As  mentioned,  further  engagement  of  psychologists  remains  essential,  particularly  as  the 
effect  of  cognitive  factors  is  a  significant  simulation  directive.  Future  research  is  encouraged  to 
engage  more  closely  with  the  psychological  community. 

5.4  Deliverables  5  &  6 

Project  Title: 

•  Deliverable  5  -  Analysis  of  the  effect  of  key  social,  cognitive,  and  informational  factors  (Apr 
2011) 

•  Deliverable  6  -  Analysis  of  the  effect  of  security  system  relationship  configurations  on  goal 
achievement  (Aug  2011),  [UNB4] 

Following  verification  and  validation  of  the  approaches  the  remaining  deliverables  centered  on 
the  development  and  implementation  of  the  extended  HSE  simulation  incorporating  feedback 
from  the  expert  discussions  into  the  models.  The  simulation  design  included  the  following 
architecture  elements:  parameter  selection  and  execution  in  a  spreadsheet  interface,  base  control 
functions  for  configuring  experiments  and  selecting  ERL  models,  the  multi-agent  simulation 
environment,  and  data  extraction  and  visualization  for  post-simulation  analysis.  These  were 
implemented  and  tested  on  core  experiments  that  varied  levels  of  the  key  social,  cognitive,  and 
informational  factors.  The  focus  on  consensus  provided  the  output  function  as  to  whether  or  not 
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the  factors  had  an  impact  on  the  achievement  of  distributed  consensus.  The  central  challenges  and 
outcomes  for  this  deliverable  are  shown  in  Table  9. 

Table  9:  Model  Refinement,  Simulation  Implementation,  and  Experiments  (Deliverables  5  &  6). 


No. 

Issues 

Outcome 

1 

•  How  to  make  a  general 
simulator? 

•  Systems  architecture 
refinement 

2 

•  How  to  allow  for 
interfacing  with  non¬ 
programmers? 

•  Using  front-end  parameter 
selection  environment 
o  Excel  spreadsheet 

3 

•  How  to  select  the  right 
tools  for  building  a 
general  simulator? 

•  Used  tools  that  provided  work 
practice  agents 

o  Other  tools  available 

4 

•  How  to  incorporate 
system  dynamic  models 
into  new  simulation? 

•  Embedded  system  dynamics 
within  agent  code  as  a  process 
o  Should  be  separated  to 
allow  for  non-code 
changes  (modularity) 

5 

•  How  to  get  more  out  of 
the  simulation  than  you 
put  into  it? 

o  How  to  avoid 
effect  of  hard¬ 
coded  solution? 

•  Stochasticity  added  to 
parameters 

o  Randomness 
o  Noise 

o  Impediments  to 
publishing  and 
observation 

o  “Goldilocks”  factor 

6 

•  How  to  test  key  factors? 
o  Social 
o  Cognitive 
o  Informational 

•  Isolating  the  effect  of  each 
factor  through  combination 
analysis 

o  Testing  all  possible 
combinations  of  the 
three  factors  by 
manipulating  whether 
or  not  the  factor  is 
randomized 

The  first  issue  involved  implementing  a  general  simulator,  and  the  outcome  was  revised 
simulation  architecture.  In  revising  the  architecture,  the  consideration  that  the  users  of  the 
simulation  environment  would  not  be  code  developers  was  a  key  concept.  This  led  to  the 
extraction  of  core  variables  from  being  hard-coded  in  the  Brahms  agent  environment  to  being 
adjustable  from  an  external  interface  (Microsoft  Excel).  The  outcome  is  a  communication 
framework  between  the  two  platforms  that  abstracts  coding  details  and  allows  for  easy  parameter 
changes.  It  also  enforces  the  requirement  that  lower-level  code  be  general  so  that  it  can  cope  with 
new  models  and  scenarios. 
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The  second  issue  relates  more  specifically  to  the  spreadsheet  interface  for  non-programmers,  as 
discussed  in  the  previous  paragraph.  The  two  are  closely  linked,  and,  as  indicated,  Microsoft 
Excel  was  selected  as  the  interface  for  users  due  to  its  familiarity  and  built-in  programming 
interface.  The  latter  proved  invaluable  for  connecting  the  spreadsheet  with  the  Brahms  agent 
environment. 

The  third  issue  continued  to  extend  the  generality  requirement  by  considering  tools  to  produce  a 
general-purpose  simulation  framework.  The  solution  was  to  continue  with  Brahms  as  a  multi¬ 
agent  platform,  but  to  add  dynamics  through  an  external  program.  This  was  satisfactorily 
achieved  during  this  phase,  and  the  combination  of  an  internal  simulator  and  an  external  variable 
controller  is  a  useful  approach. 

The  fourth  issue  involved  the  method  of  incorporating  system  dynamics  into  the  simulation.  The 
system  dynamic  models  validated  were  in  the  form  of  causal  feedback  loop  diagrams,  which 
needed  to  be  translated  into  program  variables  in  order  to  be  used.  The  process  of  making  use  of 
these  diagrams  was  a  challenge  and  the  solution  was  to  embed  the  dynamics  as  a  higher-level 
process  related  to  the  original  business  process  models.  It  is  noted  though  that  this  approach  is 
still  too  tightly  coupled  to  the  code  and  needs  to  be  generalized  further  to  allow  for  fully  changing 
these  dynamics  from  the  external  interface. 

The  fifth  issue  involved  considering  how  to  get  more  out  of  the  simulation  than  was  put  into  it, 
particularly  how  to  avoid  the  effect  of  hard-coded  results.  In  addition  to  adding  system  dynamics, 
stochastic  approaches  proved  useful,  and  randomness  was  added  to  the  parameters  based  on  the 
particular  experiment  being  run.  Noise  was  also  added  to  the  system  in  terms  of  increasing  the 
levels  of  communication  which  led  to  an  increase  in  observation  impediments  (overload),  for 
example,  and  also  using  the  “Goldilocks”  principle  for  cognitive  models,  [UNB4],  These  in  turn 
influenced  the  levels  of  information-sharing,  enabling  non-trivial  results  from  the  simulation, 
while  still  allowing  factors  to  be  tuned  in  the  spreadsheet  interface. 

The  sixth  and  final  issue  was  how  to  test  key  parameters  of  the  simulation  to  discover  the  impact 
of  the  social,  cognitive,  and  informational  factors.  The  solution  was  to  explore  these  factors 
through  combination  analysis,  specifically  by  varying  whether  or  not  each  factor  is  randomized. 
This  was  done  for  all  possible  combinations,  which  is  easily  computable  as  there  are  only  eight 
combinations  to  test  (23).  Additionally,  for  each  combination,  the  simulator  was  executed  a 
significant  number  of  times  (30)  to  produce  statistically  sound  averages.  The  lesson  learned  is  that 
this  is  an  approach  that  benefitted  highly  from  the  initial  work  in  generalizing  the  simulator  and 
its  controller,  eventually  allowing  access  to  parameters  that  coidd  be  changed  in  this  fashion. 
Moreover,  more  fine-grained  exploration  is  also  possible  for  sub  factors,  by  turning  their 
randomization  on  or  off,  rather  than  turning  the  randomization  of  whole  classes  of  factors  on  or 
off  (i.e.,  social,  cognitive,  and  informational). 

5.4.1  General  Comments 

The  overall  comment  is  that,  as  a  proof-of-concept,  the  simulation  works  well  and  allows  for 
modifying  parameters  easily.  The  use  of  the  front-end  interface  allowed  for  closing  the  loop  from 
parameter  setting  through  to  achieving  results,  without  the  user  needing  to  be  concerned  with  the 
underlying  code.  The  results  mapped  well  to  real-world  understanding  and  intuition  of  how  the 
social,  cognitive,  and  informational  factors  may  impact  a  decision-maker.  Furthermore,  there  is 
some  value  to  the  simulation  in  teaching  the  importance  of  the  cognitive  dimension  on 
collaboration.  However,  several  parameters  remain  to  be  explored  by  future  researchers, 
including  the  impact  of  need-to-share  and  need-to-know  communication  paradigms. 
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Moving  forward,  extensions  to  the  HSE  approach  should  involve  a  refinement  of  parameters  of 
interest,  an  expert  validation  of  these  refined  models  in  system  dynamics,  and  code  verification 
under  multiple  conditions  using  white-box  and  black-box  software  testing. 
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6  Conclusion 


The  Holistic  Security  Ecosystem  (HSE)  portion  of  the  T1F  project  was  tasked  with  the  goal  of 
examining  joint-organizational  systems  from  both  a  top-down  (i.e.,  individual)  and  bottom-up 
(i.e.,  system)  perspective.  It  focused  on  the  development  of  a  computer-simulation  approach  to 
explore  the  effect  of  key  social,  cognitive,  and  informational  factors  present  in  individual  agents 
(i.e.,  the  individual-level)  on  inter-organizational  efforts  (i.e.,  the  system-level),  in  particular 
consensus  achievement  through  information  sharing.  The  focus  of  the  simulation  was  on  whether 
or  not  agents  with  bounded  rationality,  only  witnessing  portions  of  the  unfolding  situation,  in 
conjunction  with  other  agents  and  based  on  the  simulation  settings;  could  collectively  agree  that  a 
threat  to  the  harbour  was  imminent.  The  experimental  results  show  that  the  cognitive  factor  has 
the  greatest  impact  on  consensus  achievement,  though  social  and  informational  factors  are  also 
important. 

The  research  to  arrive  at  the  HSE  tool  was  conducted  in  six  stages,  as  presented  in  previous 
deliverables,  and  culminates  with  a  detailed  multi-agent  simulation  involving  high-functioning 
work-practice  agents,  following  organizational  workflows,  and  being  influenced  by  social, 
cognitive,  and  informational  factors.  Such  a  simulation  can  help  to  elicit  important  information 
about  how  these  core  factors  relate  to  one  another  and  ultimately  impact  the  system.  This  suggests 
that  continued  exploration  using  computer  simulations  is  beneficial.  Additionally,  the  HSE 
approach  provides  a  core  methodology  for  constructing  such  simulations,  and  has  the  potential  to 
be  adapted  for  future  experiments  at  relatively  low  cost. 

The  lessons  learned  from  this  work  are  threefold:  (i)  more  focus  on  human  factors  through 
simulation  is  necessary  to  better  explore  the  complex  interplay  between  human  factors  and  meta- 
organizational  design;  (ii)  more  focus  on  human-factor  model  validation  is  needed  to  add 
credibility  to  the  results;  and  (iii)  more  information  on  organizational  procedures  is  needed  to 
create  meaningful  functional  models  that  can  be  “operationalized.”  These  will  lead  to  an 
improved  understanding  of  the  human  element  and  its  impact  on  joint-organizational  endeavours. 
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